The Viuva Negra crawler
نویسندگان
چکیده
This report discusses architectural aspects of web crawlers and details the design, implementation and evaluation of the Viuva Negra (VN) crawler. VN has been used for 4 years, feeding a search engine and an archive of the Portuguese web. In our experiments it crawled over 2 million documents per day, correspondent to 63 GB of data. We describe hazardous situations to crawling found on the web and the adopted solutions to mitigate their effects. The gathered information was integrated in a web warehouse that provides support for its automatic processing by text mining applications.
منابع مشابه
The Viúva Negra crawler: an experience report
This paper documents hazardous situations on the Web that crawlers must address. This knowledge was accumulated while developing and operating the Viúva Negra (VN) crawler to feed a search engine and a Web archive for the Portuguese Web for four years. The design, implementation and evaluation of the VN crawler are also presented as a case study of a Web crawler design. The case study tested pr...
متن کاملPrioritize the ordering of URL queue in Focused crawler
The enormous growth of the World Wide Web in recent years has made it necessary to perform resource discovery efficiently. For a crawler it is not an simple task to download the domain specific web pages. This unfocused approach often shows undesired results. Therefore, several new ideas have been proposed, among them a key technique is focused crawling which is able to crawl particular topical...
متن کاملNew Neandertal remains from Cova Negra (Valencia, Spain).
New Neandertal fossils from the Mousterian site of Cova Negra in the Valencia region of Spain are described, and a comprehensive study of the entire human fossil sample is provided. The new specimens significantly augment the sample of human remains from this site and make Cova Negra one of the richest human paleontological sites on the Iberian Peninsula. The new specimens include cranial and p...
متن کاملAssessment of Crawler Tractor Effects on Soil Surface Properties
Skidding operations can cause considerable and wide spread soil disturbance. The objective of this study was to evaluate the effects of ground skidding operations on soil compaction, moisture content, and total porosity at different levels of slope and traffic frequency. Four levels of traffic intensity (3, 8, 13, and >13 passes of a Crawler Tractor Onezhets ? 110) and three levels of slope ( ...
متن کاملAnalytical characterization of the aroma of Tinta Negra Mole red wine: Identification of the main odorants compounds
A method for the simultaneous determination of major and minor volatiles composition in different types (dry, medium dry, sweet and medium sweet) of a young Tinta Negra Mole (TNM) monovarietal red wine from 2003 harvest has been validated. Wine samples preparation includes a dichloromethane liquid–liquid extraction followed by concentration under a nitrogen atmosphere. The extracted fraction wa...
متن کامل